Computers Seeing Action 1 Seeing Action

As research in computer vision has shifted from only processing single, static images to the manipulation of video sequences, the concept of action recognition has become important. Fundamental to understanding action is reasoning about time, in either an implicit or explicit framework. In this paper I describe several spe-ciic examples of incorporating time into representations of action and how those representations are used to recognize actions. The approaches diier on whether variation over time is considered a continuous mapping, a state-based trajectory, or a qualitative, semantically labeled sequence. For two of the domains | whole body actions and hand gestures | I describe the approaches in detail while two others | constrained semantic domains (e.g. watching someone cooking) and labeling dynamic events (e.g. American football) | are brieey mentioned. Understanding video sequences is diierent than conventional image understanding in that one is interested in what is happening in a scene, as opposed to what is in the scene. One might believe that attempting to describe what is happening in hundreds of images is not a viable research of goal given the diiculty of understanding just one picture. However, video understanding can be regarded as a way of providing more constraint in the interpretation of imagery. We require that the image interpretation be plausible over time: extracted structure must obey the temporal constraints of the domain. For example, if we are annotating an American football play, we might be interested in tracking the quarterback. Unfortunately, current (even near future) technology cannot see or track the quarterback in every frame. However, assuming he never disappears from the eld of play, we can \track" him as he enters an amorphous blob and re-emerges six frames later. The program cannot see him during this time, but it knows he's there. Understanding time can be either explicit, as in the above example, or implicit, captured in the representation of action. One example that we will expand upon later is our work in gesture recognition 3, 22]. In this work gesture is represented either deterministically by an explicit sequence of states through which the hand must move, or probabilistically by a hidden Markov model. In both cases the requirement that the interpretation be consistent with the temporal constraints of the domain is guaranteed by matching the input data to learned representations of action which are sensitive to time. From our perspective, one of the future directions …